Fix kusama validators getting 0 backing rewards the first session they enter the active set #3722
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…y enter active set Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
bkchr
approved these changes
Mar 17, 2024
Co-authored-by: Bastian Köcher <git@kchr.de>
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
I agree the fix is pretty clear, I just want to make sure there aren't other places that I missed. We don't need to wait with merging this PR, until I get that setup working.
Done.
Did my best hopefully it is clearer now. |
s0me0ne-unkn0wn
approved these changes
Mar 18, 2024
There was a problem hiding this comment.
Looks good! I think, although it would be good to test on Versi, it's not strictly required. By the nature of changes, the worst possible scenario is that we're updating something we shouldn't update with the data still correct, so it doesn't make any worse anyway.
If you want to test this, run a test network locally. Set |
lexnv
approved these changes
Mar 18, 2024
bkchr
added
T1-FRAME
|
@alexggh please add some small prdoc explaining the change. |
|
Does this also fix #1819? |
It does help, statement-v1 does not require direct connections to your peers in the validator group, so you might get lucky with the random gossiping when you are sparsely connected, but fully lucky, so I guess that's why we would see just some missed statements instead of missing all the statements as it happens with statements-v2. |
sandreim
approved these changes
Mar 18, 2024
Yeah, that's what I ended-up doing, it is not that easy, because ValidatorManager changes the set every session, so it was masking the problem also you need some bigger waiting times and sessions because DHT is updated every 10 minutes. Managed to reproduce it with zombienet and a modified ValidatorManager and confirmed this patch is fixing the issue. |
Done. |
alexggh
added a commit
that referenced
this pull request
Mar 18, 2024
…y enter the active set (#3722) There is a problem in the way we update `authorithy-discovery` next keys and because of that nodes that enter the active set would be noticed at the start of the session they become active, instead of the start of the previous session as it was intended. This is problematic because: 1. The node itself advertises its addresses on the DHT only when it notices it should become active on around ~10m loop, so in this case it would notice after it becomes active. 2. The other nodes won't be able to detect the new nodes addresses at the beginning of the session, so it won't added them to the reserved set. With 1 + 2, we end-up in a situation where the the new node won't be able to properly connect to its peers because it won't be in its peers reserved set. Now, the nodes accept by default`MIN_GOSSIP_PEERS: usize = 25` connections to nodes that are not in the reserved set, but given Kusama size(> 1000 nodes) you could easily have more than`25` new nodes entering the active set or simply the nodes don't have slots anymore because, they already have connections to peers not in the active set. In the end what the node would notice is 0 backing rewards because it wasn't directly connected to the peers in its backing group. ## Root-cause The flow is like this: 1. At BAD_SESSION - 1, in `rotate_session` new nodes are added to QueuedKeys https://github.com/paritytech/polkadot-sdk/blob/02e1a7f476d7d7c67153e975ab9a1bdc02ffea12/substrate/frame/session/src/lib.rs#L609 ``` <QueuedKeys<T>>::put(queued_amalgamated.clone()); <QueuedChanged<T>>::put(next_changed); ``` 2. AuthorityDiscovery::on_new_session is called with `changed` being the value of `<QueuedChanged<T>>:` at BAD_SESSION - **2** because it was saved before being updated https://github.com/paritytech/polkadot-sdk/blob/02e1a7f476d7d7c67153e975ab9a1bdc02ffea12/substrate/frame/session/src/lib.rs#L613 3. At BAD_SESSION - 1, `AuthorityDiscovery::on_new_session` doesn't updated its next_keys because `changed` was false. 4. For the entire durations of `BAD_SESSION - 1` everyone calling runtime api `authorities`(should return past, present and future authorities) won't discover the nodes that should become active . 5. At the beginning of BAD_SESSION, all nodes discover the new nodes are authorities, but it is already too late because reserved_nodes are updated only at the beginning of the session by the `gossip-support`. See above why this bad. ## Fix Update next keys with the queued_validators at every session, not matter the value of `changed` this is the same way babe pallet correctly does it. https://github.com/paritytech/polkadot-sdk/blob/02e1a7f476d7d7c67153e975ab9a1bdc02ffea12/substrate/frame/babe/src/lib.rs#L655 ## Notes - The issue doesn't reproduce with proof-authorities changes like `versi` because `changed` would always be true and `AuthorityDiscovery` correctly updates its next_keys every time. - Confirmed at session `37651` on kusama that this is exactly what it happens by looking at blocks with polkadot.js. ## TODO - [ ] Move versi on proof of stake and properly test before and after fix to confirm there is no other issue. --------- Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io> Co-authored-by: Bastian Köcher <git@kchr.de> (cherry picked from commit 8d0cd4f) Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
alexggh
added a commit
that referenced
this pull request
Mar 18, 2024
…y enter the active set (#3722) There is a problem in the way we update `authorithy-discovery` next keys and because of that nodes that enter the active set would be noticed at the start of the session they become active, instead of the start of the previous session as it was intended. This is problematic because: 1. The node itself advertises its addresses on the DHT only when it notices it should become active on around ~10m loop, so in this case it would notice after it becomes active. 2. The other nodes won't be able to detect the new nodes addresses at the beginning of the session, so it won't added them to the reserved set. With 1 + 2, we end-up in a situation where the the new node won't be able to properly connect to its peers because it won't be in its peers reserved set. Now, the nodes accept by default`MIN_GOSSIP_PEERS: usize = 25` connections to nodes that are not in the reserved set, but given Kusama size(> 1000 nodes) you could easily have more than`25` new nodes entering the active set or simply the nodes don't have slots anymore because, they already have connections to peers not in the active set. In the end what the node would notice is 0 backing rewards because it wasn't directly connected to the peers in its backing group. ## Root-cause The flow is like this: 1. At BAD_SESSION - 1, in `rotate_session` new nodes are added to QueuedKeys https://github.com/paritytech/polkadot-sdk/blob/02e1a7f476d7d7c67153e975ab9a1bdc02ffea12/substrate/frame/session/src/lib.rs#L609 ``` <QueuedKeys<T>>::put(queued_amalgamated.clone()); <QueuedChanged<T>>::put(next_changed); ``` 2. AuthorityDiscovery::on_new_session is called with `changed` being the value of `<QueuedChanged<T>>:` at BAD_SESSION - **2** because it was saved before being updated https://github.com/paritytech/polkadot-sdk/blob/02e1a7f476d7d7c67153e975ab9a1bdc02ffea12/substrate/frame/session/src/lib.rs#L613 3. At BAD_SESSION - 1, `AuthorityDiscovery::on_new_session` doesn't updated its next_keys because `changed` was false. 4. For the entire durations of `BAD_SESSION - 1` everyone calling runtime api `authorities`(should return past, present and future authorities) won't discover the nodes that should become active . 5. At the beginning of BAD_SESSION, all nodes discover the new nodes are authorities, but it is already too late because reserved_nodes are updated only at the beginning of the session by the `gossip-support`. See above why this bad. ## Fix Update next keys with the queued_validators at every session, not matter the value of `changed` this is the same way babe pallet correctly does it. https://github.com/paritytech/polkadot-sdk/blob/02e1a7f476d7d7c67153e975ab9a1bdc02ffea12/substrate/frame/babe/src/lib.rs#L655 ## Notes - The issue doesn't reproduce with proof-authorities changes like `versi` because `changed` would always be true and `AuthorityDiscovery` correctly updates its next_keys every time. - Confirmed at session `37651` on kusama that this is exactly what it happens by looking at blocks with polkadot.js. ## TODO - [ ] Move versi on proof of stake and properly test before and after fix to confirm there is no other issue. --------- Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io> Co-authored-by: Bastian Köcher <git@kchr.de> (cherry picked from commit 8d0cd4f) Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
alexggh
added a commit
that referenced
this pull request
Mar 18, 2024
…y enter the active set (#3722) There is a problem in the way we update `authorithy-discovery` next keys and because of that nodes that enter the active set would be noticed at the start of the session they become active, instead of the start of the previous session as it was intended. This is problematic because: 1. The node itself advertises its addresses on the DHT only when it notices it should become active on around ~10m loop, so in this case it would notice after it becomes active. 2. The other nodes won't be able to detect the new nodes addresses at the beginning of the session, so it won't added them to the reserved set. With 1 + 2, we end-up in a situation where the the new node won't be able to properly connect to its peers because it won't be in its peers reserved set. Now, the nodes accept by default`MIN_GOSSIP_PEERS: usize = 25` connections to nodes that are not in the reserved set, but given Kusama size(> 1000 nodes) you could easily have more than`25` new nodes entering the active set or simply the nodes don't have slots anymore because, they already have connections to peers not in the active set. In the end what the node would notice is 0 backing rewards because it wasn't directly connected to the peers in its backing group. ## Root-cause The flow is like this: 1. At BAD_SESSION - 1, in `rotate_session` new nodes are added to QueuedKeys https://github.com/paritytech/polkadot-sdk/blob/02e1a7f476d7d7c67153e975ab9a1bdc02ffea12/substrate/frame/session/src/lib.rs#L609 ``` <QueuedKeys<T>>::put(queued_amalgamated.clone()); <QueuedChanged<T>>::put(next_changed); ``` 2. AuthorityDiscovery::on_new_session is called with `changed` being the value of `<QueuedChanged<T>>:` at BAD_SESSION - **2** because it was saved before being updated https://github.com/paritytech/polkadot-sdk/blob/02e1a7f476d7d7c67153e975ab9a1bdc02ffea12/substrate/frame/session/src/lib.rs#L613 3. At BAD_SESSION - 1, `AuthorityDiscovery::on_new_session` doesn't updated its next_keys because `changed` was false. 4. For the entire durations of `BAD_SESSION - 1` everyone calling runtime api `authorities`(should return past, present and future authorities) won't discover the nodes that should become active . 5. At the beginning of BAD_SESSION, all nodes discover the new nodes are authorities, but it is already too late because reserved_nodes are updated only at the beginning of the session by the `gossip-support`. See above why this bad. ## Fix Update next keys with the queued_validators at every session, not matter the value of `changed` this is the same way babe pallet correctly does it. https://github.com/paritytech/polkadot-sdk/blob/02e1a7f476d7d7c67153e975ab9a1bdc02ffea12/substrate/frame/babe/src/lib.rs#L655 ## Notes - The issue doesn't reproduce with proof-authorities changes like `versi` because `changed` would always be true and `AuthorityDiscovery` correctly updates its next_keys every time. - Confirmed at session `37651` on kusama that this is exactly what it happens by looking at blocks with polkadot.js. ## TODO - [ ] Move versi on proof of stake and properly test before and after fix to confirm there is no other issue. --------- Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io> Co-authored-by: Bastian Köcher <git@kchr.de> (cherry picked from commit 8d0cd4f) Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
alexggh
added a commit
that referenced
this pull request
Mar 18, 2024
…y enter the active set (#3722) There is a problem in the way we update `authorithy-discovery` next keys and because of that nodes that enter the active set would be noticed at the start of the session they become active, instead of the start of the previous session as it was intended. This is problematic because: 1. The node itself advertises its addresses on the DHT only when it notices it should become active on around ~10m loop, so in this case it would notice after it becomes active. 2. The other nodes won't be able to detect the new nodes addresses at the beginning of the session, so it won't added them to the reserved set. With 1 + 2, we end-up in a situation where the the new node won't be able to properly connect to its peers because it won't be in its peers reserved set. Now, the nodes accept by default`MIN_GOSSIP_PEERS: usize = 25` connections to nodes that are not in the reserved set, but given Kusama size(> 1000 nodes) you could easily have more than`25` new nodes entering the active set or simply the nodes don't have slots anymore because, they already have connections to peers not in the active set. In the end what the node would notice is 0 backing rewards because it wasn't directly connected to the peers in its backing group. ## Root-cause The flow is like this: 1. At BAD_SESSION - 1, in `rotate_session` new nodes are added to QueuedKeys https://github.com/paritytech/polkadot-sdk/blob/02e1a7f476d7d7c67153e975ab9a1bdc02ffea12/substrate/frame/session/src/lib.rs#L609 ``` <QueuedKeys<T>>::put(queued_amalgamated.clone()); <QueuedChanged<T>>::put(next_changed); ``` 2. AuthorityDiscovery::on_new_session is called with `changed` being the value of `<QueuedChanged<T>>:` at BAD_SESSION - **2** because it was saved before being updated https://github.com/paritytech/polkadot-sdk/blob/02e1a7f476d7d7c67153e975ab9a1bdc02ffea12/substrate/frame/session/src/lib.rs#L613 3. At BAD_SESSION - 1, `AuthorityDiscovery::on_new_session` doesn't updated its next_keys because `changed` was false. 4. For the entire durations of `BAD_SESSION - 1` everyone calling runtime api `authorities`(should return past, present and future authorities) won't discover the nodes that should become active . 5. At the beginning of BAD_SESSION, all nodes discover the new nodes are authorities, but it is already too late because reserved_nodes are updated only at the beginning of the session by the `gossip-support`. See above why this bad. ## Fix Update next keys with the queued_validators at every session, not matter the value of `changed` this is the same way babe pallet correctly does it. https://github.com/paritytech/polkadot-sdk/blob/02e1a7f476d7d7c67153e975ab9a1bdc02ffea12/substrate/frame/babe/src/lib.rs#L655 ## Notes - The issue doesn't reproduce with proof-authorities changes like `versi` because `changed` would always be true and `AuthorityDiscovery` correctly updates its next_keys every time. - Confirmed at session `37651` on kusama that this is exactly what it happens by looking at blocks with polkadot.js. ## TODO - [ ] Move versi on proof of stake and properly test before and after fix to confirm there is no other issue. --------- Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io> Co-authored-by: Bastian Köcher <git@kchr.de> (cherry picked from commit 8d0cd4f) Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
alexggh
added a commit
that referenced
this pull request
Mar 18, 2024
…y enter the active set (#3722) There is a problem in the way we update `authorithy-discovery` next keys and because of that nodes that enter the active set would be noticed at the start of the session they become active, instead of the start of the previous session as it was intended. This is problematic because: 1. The node itself advertises its addresses on the DHT only when it notices it should become active on around ~10m loop, so in this case it would notice after it becomes active. 2. The other nodes won't be able to detect the new nodes addresses at the beginning of the session, so it won't added them to the reserved set. With 1 + 2, we end-up in a situation where the the new node won't be able to properly connect to its peers because it won't be in its peers reserved set. Now, the nodes accept by default`MIN_GOSSIP_PEERS: usize = 25` connections to nodes that are not in the reserved set, but given Kusama size(> 1000 nodes) you could easily have more than`25` new nodes entering the active set or simply the nodes don't have slots anymore because, they already have connections to peers not in the active set. In the end what the node would notice is 0 backing rewards because it wasn't directly connected to the peers in its backing group. ## Root-cause The flow is like this: 1. At BAD_SESSION - 1, in `rotate_session` new nodes are added to QueuedKeys https://github.com/paritytech/polkadot-sdk/blob/02e1a7f476d7d7c67153e975ab9a1bdc02ffea12/substrate/frame/session/src/lib.rs#L609 ``` <QueuedKeys<T>>::put(queued_amalgamated.clone()); <QueuedChanged<T>>::put(next_changed); ``` 2. AuthorityDiscovery::on_new_session is called with `changed` being the value of `<QueuedChanged<T>>:` at BAD_SESSION - **2** because it was saved before being updated https://github.com/paritytech/polkadot-sdk/blob/02e1a7f476d7d7c67153e975ab9a1bdc02ffea12/substrate/frame/session/src/lib.rs#L613 3. At BAD_SESSION - 1, `AuthorityDiscovery::on_new_session` doesn't updated its next_keys because `changed` was false. 4. For the entire durations of `BAD_SESSION - 1` everyone calling runtime api `authorities`(should return past, present and future authorities) won't discover the nodes that should become active . 5. At the beginning of BAD_SESSION, all nodes discover the new nodes are authorities, but it is already too late because reserved_nodes are updated only at the beginning of the session by the `gossip-support`. See above why this bad. ## Fix Update next keys with the queued_validators at every session, not matter the value of `changed` this is the same way babe pallet correctly does it. https://github.com/paritytech/polkadot-sdk/blob/02e1a7f476d7d7c67153e975ab9a1bdc02ffea12/substrate/frame/babe/src/lib.rs#L655 ## Notes - The issue doesn't reproduce with proof-authorities changes like `versi` because `changed` would always be true and `AuthorityDiscovery` correctly updates its next_keys every time. - Confirmed at session `37651` on kusama that this is exactly what it happens by looking at blocks with polkadot.js. ## TODO - [ ] Move versi on proof of stake and properly test before and after fix to confirm there is no other issue. --------- Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io> Co-authored-by: Bastian Köcher <git@kchr.de> (cherry picked from commit 8d0cd4f) Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
2 tasks
dharjeezy
pushed a commit
to dharjeezy/polkadot-sdk
that referenced
this pull request
Mar 24, 2024
…y enter the active set (paritytech#3722) There is a problem in the way we update `authorithy-discovery` next keys and because of that nodes that enter the active set would be noticed at the start of the session they become active, instead of the start of the previous session as it was intended. This is problematic because: 1. The node itself advertises its addresses on the DHT only when it notices it should become active on around ~10m loop, so in this case it would notice after it becomes active. 2. The other nodes won't be able to detect the new nodes addresses at the beginning of the session, so it won't added them to the reserved set. With 1 + 2, we end-up in a situation where the the new node won't be able to properly connect to its peers because it won't be in its peers reserved set. Now, the nodes accept by default`MIN_GOSSIP_PEERS: usize = 25` connections to nodes that are not in the reserved set, but given Kusama size(> 1000 nodes) you could easily have more than`25` new nodes entering the active set or simply the nodes don't have slots anymore because, they already have connections to peers not in the active set. In the end what the node would notice is 0 backing rewards because it wasn't directly connected to the peers in its backing group. ## Root-cause The flow is like this: 1. At BAD_SESSION - 1, in `rotate_session` new nodes are added to QueuedKeys https://github.com/paritytech/polkadot-sdk/blob/02e1a7f476d7d7c67153e975ab9a1bdc02ffea12/substrate/frame/session/src/lib.rs#L609 ``` <QueuedKeys<T>>::put(queued_amalgamated.clone()); <QueuedChanged<T>>::put(next_changed); ``` 2. AuthorityDiscovery::on_new_session is called with `changed` being the value of `<QueuedChanged<T>>:` at BAD_SESSION - **2** because it was saved before being updated https://github.com/paritytech/polkadot-sdk/blob/02e1a7f476d7d7c67153e975ab9a1bdc02ffea12/substrate/frame/session/src/lib.rs#L613 3. At BAD_SESSION - 1, `AuthorityDiscovery::on_new_session` doesn't updated its next_keys because `changed` was false. 4. For the entire durations of `BAD_SESSION - 1` everyone calling runtime api `authorities`(should return past, present and future authorities) won't discover the nodes that should become active . 5. At the beginning of BAD_SESSION, all nodes discover the new nodes are authorities, but it is already too late because reserved_nodes are updated only at the beginning of the session by the `gossip-support`. See above why this bad. ## Fix Update next keys with the queued_validators at every session, not matter the value of `changed` this is the same way babe pallet correctly does it. https://github.com/paritytech/polkadot-sdk/blob/02e1a7f476d7d7c67153e975ab9a1bdc02ffea12/substrate/frame/babe/src/lib.rs#L655 ## Notes - The issue doesn't reproduce with proof-authorities changes like `versi` because `changed` would always be true and `AuthorityDiscovery` correctly updates its next_keys every time. - Confirmed at session `37651` on kusama that this is exactly what it happens by looking at blocks with polkadot.js. ## TODO - [ ] Move versi on proof of stake and properly test before and after fix to confirm there is no other issue. --------- Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io> Co-authored-by: Bastian Köcher <git@kchr.de>
This was referenced Jun 5, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There is a problem in the way we update
authorithy-discoverynext keys and because of that nodes that enter the active set would be noticed at the start of the session they become active, instead of the start of the previous session as it was intended. This is problematic because:With 1 + 2, we end-up in a situation where the the new node won't be able to properly connect to its peers because it won't be in its peers reserved set. Now, the nodes accept by default
MIN_GOSSIP_PEERS: usize = 25connections to nodes that are not in the reserved set, but given Kusama size(> 1000 nodes) you could easily have more than25new nodes entering the active set or simply the nodes don't have slots anymore because, they already have connections to peers not in the active set.In the end what the node would notice is 0 backing rewards because it wasn't directly connected to the peers in its backing group.
Root-cause
The flow is like this:
rotate_sessionnew nodes are added to QueuedKeyspolkadot-sdk/substrate/frame/session/src/lib.rs
Line 609 in 02e1a7f
changedbeing the value of<QueuedChanged<T>>:at BAD_SESSION - 2 because it was saved before being updatedpolkadot-sdk/substrate/frame/session/src/lib.rs
Line 613 in 02e1a7f
AuthorityDiscovery::on_new_sessiondoesn't updated its next_keys becausechangedwas false.BAD_SESSION - 1everyone calling runtime apiauthorities(should return past, present and future authorities) won't discover the nodes that should become active .gossip-support. See above why this bad.Fix
Update next keys with the queued_validators at every session, not matter the value of
changedthis is the same way babe pallet correctly does it.polkadot-sdk/substrate/frame/babe/src/lib.rs
Line 655 in 02e1a7f
Notes
versibecausechangedwould always be true andAuthorityDiscoverycorrectly updates its next_keys every time.37651on kusama that this is exactly what it happens by looking at blocks with polkadot.js.TODO